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Abstract 

In the media literacy literature, the challenges associated with assessment have, to a great extent, been ignored. The 
purpose of this mixed methods study was therefore to explore the views of media literacy scholars and professionals on 
assessment challenges through qualitative interviews (n = 10) with the intent of using this information to develop a 
quantitative survey to validate and extend the qualitative findings with a larger sample of media literacy professionals 
and scholars from around the world (n = 133). The findings offer an overview of the assessment challenges encountered 
by these participants. 
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It's the age-old question: How do we oppose prejudice 
without exhibiting prejudice against people who are 
prejudiced? In teaching, how do we assess how open-minded 
someone is without ourselves being closed-minded about 
closed-mindedness? 

- Mr. Green, interviewee in this study 

Media literacy education (MLE) has garnered much attention recently. The importance of (digital) 
media literacy skills is mentioned in the higher education edition of the latest New Media Consortium Horizon 
Report (Johnson, Adams Becker, Estrada, and Freeman 2014). It is also acknowledged by organizations such 
as the Association of College & Research Libraries (2012) and the Partnership for 21st Century Skills (2011), 
who advocate for the inclusion of media literacy skills in their vision and frameworks for student success in 
contemporary society. Furthermore, Hobbs (2011) argues that there is more engaged participation between 
scholars and practitioners on media literacy than was ever dreamed of just a decade ago. 

Even though MLE seems to be thriving, media literacy assessment remains an issue of concern. 
According to Livingstone and Thumim (2003), there is little consensus over the appropriate way to measure 
media literacy. This is reflected in the variety of ways media literacy is assessed. Livingstone and Thumim 
(2003) state that different research methodologies and samples make it difficult to draw comparisons. Scharrer 
(2002) argues that even though there is a generalized understanding about what media literacy outcomes are, 
they are often not explicitly defined and measured. Similarly, Bergsma, and Carney (2008) believe that media 
literacy scholars and professionals should be more precise about the concepts and skills to include in their 
lessons. Media literacy should be more clearly defined and standards or criteria to assess it should be 
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developed (Christ 2004; Zacchetti 2011). Martens (2010) states that evaluating and explaining the 
effectiveness of MLE is one of the most overwhelming challenges to be addressed by research in the field. 

Though it is often asserted in the literature that the development of media literacy assessment is 
challenging (see: Livingstone and Thumim 2003; Martens 2010; Scharrer 2002; Scheibe and Rogow 2011), 
little systematic research has been conducted on this topic. Although the literature provides anecdotal evidence 
on media literacy challenges, less is known about the challenges that are commonly encountered by media 
literacy scholars and professionals who work in the field. A better understanding of the challenges surrounding 
media literacy assessment may provide a starting point for policy makers, scholars, educators, and other media 
literacy practitioners to address these challenges. 

This study provides a detailed account of the challenges that professionals identify regarding media 
literacy assessment. I am to explore media literacy assessment challenges as experienced by media literacy 
professionals and scholars through qualitative interviews with the intent of using qualitative data to develop a 
quantitative survey to validate and extend the findings with a larger sample. The research question is: Which 
challenges do media literacy professionals and scholars identify regarding media literacy assessment? 

Review of Literature 


Media Literacy Assessment 

Educational assessment is “a process for obtaining information that is used for making decisions about 
students; curricula, programs, and schools; and educational policy” (Nitko and Brookhart 2011, 3). A variety 
of entities have assessed MLE over time. For example, media literacy scholars often assess student 
achievement in order to evaluate the effectiveness of MLE. In addition, several countries, states, and 
organizations assess media literacy on a broader scale to get a sense of how media literate their citizens or 
groups of students in their states or countries are. Finally, individual media educators assess students’ media 
literacy skills in their classrooms, after-school classes, libraries, or other settings for a variety of purposes. 

Researchers who have attempted to assess media literacy come from various fields in which a variety of 
approaches towards assessment are applied. MLE generally appears to be assessed in three different ways by 
media literacy scholars. First, MLE is occasionally assessed by measuring separate constructs or outcomes that 
do not directly relate to media literacy knowledge, skills, and attitudes. The examples include measuring the 
effectiveness of MLE by assessing students’ willingness to use aggression (Byme 2005), risk factors of eating 
disorders (Wade, Davidson, and O’Dea 2003), or by measuring constructs such as children's food and 
vegetable intake (Evans et al. 2006). Second, media literacy is at times measured by assessing selective 
components, such as perception of bias (Vraga et al. 2009), perceived realism (Austin and Johnson 1995; 

Irving and Berel 2001; Kusel 1999; Pinkleton et al. 2007; Vooijs and van der Voort 1993), or response to 
media messages related to smoking (Primack et al. 2006; Primack et al. 2009). 

Some researchers and organizations have developed instruments to assess media literacy more 
holistically (e.g. Arke and Primack 2009; Chang and Lui 2011; Duran et al. 2008; EAVI 2010, 2011; Hobbs 
and Frost 1998, 2003; Inan and Temur 2012; Maksl, Ashley, and Craft 2015; Quin and McMahon 1993; 
UNESCO 2013; Worsnop 1996; Wulff 1997). Most of these instruments are quantitative in nature, though 
some researchers also employ qualitative measures. Even though in most cases media literacy is assessed as a 
whole, researchers usually target specific populations (such as high school students, adults, or teachers) and 
specific types of media messages (like advertisements or the news). Many of the quantitative measures include 
self-assessment measures (Chang and Lui 2011; EAVI 2010, 2011; Inan and Temur 2012). In these types of 
assessments, the participants or respondents are asked to rate their own knowledge and skills. 

MLE is not only assessed by researchers and organizations. It has also been assessed in educational 
systems, states, and countries across the globe. In many different countries, MLE has been integrated in the 
curriculum or taught as a separate subject in formal school systems. Although it is taught in many different 
school systems across the world, it is only a substantial and assessed part of the school system in a few 
countries (Buckingham and Domaille 2009). It is assessed on a national level in countries such as New Zealand 
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(Lealand 2009) and the United Kingdom (British Film Institute 2013). In some countries such as the United 
States, Canada, and Australia it may be measured on a state or province-wide level. In these countries there are 
wide differences to the extent media literacy is part of the curriculum; this affects the extent to which it is 
assessed. For example, although Australia is possibly the most developed when it comes to MLE, it is assessed 
differently in different states. In addition, while MLE is assessed in a few provinces and states in Canada and 
the US, it is not taught and assessed in all provinces or states in these countries. The European MLE Study 
(EMEDUS 2014) recently published 27 reports on fonnal MLE and assessment for 27 European countries. 

According to EMEDUS (2014), in most countries media literacy is integrated into different subject 
areas in a cross-curricular way, rather than treated as a separate subject. Examples of countries that have 
integrated media literacy in a cross-curricular way are: Austria, Belgium, Bulgaria, Czech Republic, Estonia, 
France, Finland, Gennany, Hungary, Ireland, Italy, Latvia, Luxemburg, Malta, the Netherlands, Poland, 
Portugal, Slovenia, Slovakia, Spain, and Sweden. Although some of these countries, such as Bulgaria, Slovenia 
and Slovakia, treat media literacy as a compulsory cross-disciplinary subject, others such as France and 
Sweden only loosely refer to media literacy in their curriculum (EMEDUS 2014). Interestingly, while many 
countries teach media literacy in a cross-curricular way, media literacy outcomes are not assessed at all in most 
of these countries (EMEDUS 2014). 

Media Literacy Assessment Challenges 

A lack of systematic implementation of media literacy assessments is one of the challenges that the 
field faces. Although many countries in the world require media literacy to be taught in their curricula, in most 
places there is a lack of any clearly assessed activities measuring student learning in terms of skills and 
competencies (Buckingham and Domaille, 2009). Luckily, some organizations such as the Association for 
Media Literacy (AML) in Canada and the British Film Institute (BFI) in England have developed media 
literacy frameworks and a few countries (like the UK, Australia, and New Zealand) have developed 
specifications of media literacy skills and competencies for in their curricula (Buckingham and Domaille, 
2009). However, respondents of Buckingham and Domaille’s (2009) international survey conducted for 
UNESCO indicated that media education is often included in curriculum documents but that it is not assessed 
by itself (or assessed at all). Overall, Buckingham and Domaille (2009) claimed that that the lack of structured 
assessment procedures likely contributed to a lack of status of MLE. They claim that the field has struggled for 
recognition and has mainly been included within assessment of other subject areas. 

A challenge that may relate to this lack of systematic implementation of media literacy assessments 
across different educational systems is that media literacy criteria and outcomes are not always clearly defined. 
For example, Christ (2004) claims that the term media literacy should be more clearly defined and that 
standards and competencies need to be developed to measure media literacy outcomes. He states that most 
higher education faculty would claim that they teach media literacy however, they may not be able to express 
what they mean with regard to the term and much less be able to assess it with learning outcomes (Christ 
2004). Similarly, Bergsma and Camey (2008, 522) suggest that media literacy professionals and scholars 
“should be more explicit about the media literacy core concepts/skills they include in their interventions, and 
should more carefully address who delivered the intervention with what fidelity, in what setting, for how long 
and utilizing what pedagogical approach.” 

On another note, EAVI (2010, 2011) claims that refining the criteria or indicators of media literacy is 
challenging as the concept relates to many different contexts, actions, and influences. They assert that a single 
survey cannot provide a comprehensive assessment of media literacy and that other approaches are needed. 
According to Schwarz et al. (2011), media literacy is so multidimensional and complex that it is practically 
impossible to develop one instrument that could assess all aspects of media literacy and all target groups. Their 
suggestion is developing assessments for specific components of media literacy and specific target groups. 
Media literacy is embedded in many different subject areas and contexts, which makes a uniform set of 
assessment instruments impossible (Scheibe and Rogow, 2011). Although UNESCO (2013) recently 
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developed a competency matrix for media and information literacy, actual assessment instruments for this 
matrix have yet to be developed. 

Martens (2010, 15) argues that “researchers should urgently develop more valid and reliable research 
instruments to aptly capture media learning outcomes.” He suggests that researchers examine whether the skills 
learned in MLE transfers to new situations. He doubts that many of the results of experimental research 
generalize to everyday media use and suggests that new research should aim to capture the long-tenn influence 
of MLE on individuals’ daily life. 

Fastrez (2009, 3) claims that media literacy assessment is “still in its infancy” and “lacks systematic 
efforts to make it a coherent endeavor.” Even though several assessment frameworks and instruments have 
appeared over the past two decades (e.g. Arke and Primack 2009; EAVI 2010, 2011; Quin and McMahon 
1995; Worsnop 1996; Wulff 1997), they still have to be examined further for validity and reliability (Primack 
et al. 2006; Primack et al. 2009). Possible disadvantages with self-assessment have also not been addressed. 
Even though self-assessment seems common (e.g. used by Chang and Lui 2011; EAVI 2010, 2011; Inan and 
Temur 2012), it is typically not regarded as a valid measure of acquired skills for a few reasons. Dunning, 
Heath, and Suls (2004) argue that the correlation between self-ratings of skill and actual performance is 
moderate to small in many domains. They contend that students tend to overrate themselves, claiming that they 
are above average in a skill. In addition, students overestimate the likelihood that they will engage in desirable 
behaviors and achieve favorable outcomes (Dunning, Heath, and Suls 2004). Media literacy self-assessments 
may therefore measure people’s confidence surrounding their use, evaluation, and creation of media messages, 
rather than their actual competence. 

In the context of communication research, MLE is generally assessed quantitatively by researchers who 
test interventions in controlled settings as a means to mitigate harmful effects. MLE in field-based, naturalistic 
classroom environments is generally assessed qualitatively by scholars adopting a critical/cultural studies 
approach (Martens 2010). Interestingly, quantitative research and a more defensive approach largely dominates 
the research agenda (Martens 2010). According to the Center for Media Literacy (CML n.d.), assessment 
usually focuses on tests related to lower-order knowledge and skills that multiple choice questions foster, and 
efforts to broaden assessments to include higher order skills is challenging and ongoing (CML n.d.). According 
to Scheibe and Rogow (2011) assessment would be easier if MLE was about teaching discrete facts. It is more 
difficult to assess the process of asking questions and reflecting on production choices when a variety of 
responses is the nonn (Scheibe and Rogow 2011). Cheng (2009) describes the majority of media production 
work as creative in nature, and therefore not particularly conducive to standardized means of assessments. 
Although higher order thinking skills such as analyzing, evaluating, and producing media messages is integral 
to MLE, assessment of these skills appear challenging and they do not always appear to be assessed. Several 
challenges related to media literacy assessment have been identified in the literature. In the next section, I 
examine the perceptions of assessment challenges by offering perspectives from media literacy educators and 
researchers from several countries. 


Methodology 

This study employed mixed methods, combining qualitative and quantitative approaches in order to 
identify the challenges professionals and scholars identify regarding media literacy assessment. An exploratory 
approach was applied in two phases. First, qualitative interview data with a small sample of MLE experts were 
collected and analyzed to explore the challenges of media literacy assessment with the intent of using this 
information to develop a quantitative survey to validate and extend the qualitative findings with a larger 
sample of media literacy professionals and scholars from around the world. Qualitative data were obtained 
from media literacy professionals through interviews (n = 10). Then a more descriptive view of the issues and 
challenges of MLE was advanced by the administration of a quantitative survey (n = 133). 

This study was part of a larger research study on perceptions of media literacy assessment. In addition 
to questions about assessment challenges, the interviews and surveys also included questions on media literacy 
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goals, assessment instruments, and recommendations that media literacy professionals and scholars had 
regarding media literacy assessment. As the entire study would be too voluminous to be discussed in a single 
article, this paper only discusses the issues and challenges of MLE in detail. 

Phase 1: Qualitative Interviews 

Participants. Qualitative studies typically focus in depth on a relatively small sample that is selected 
purposefully. Patton (2002) stated that “the logic and power of purposive sampling lie in selecting information 
rich cases for study in depth” (230). For this reason, information rich cases were selected for the qualitative 
interviews. The specific strategy used to select participants was a combination of extreme case sampling, by 
selecting information-rich cases with much knowledge on the topic, and maximum variation sampling, by 
selecting media literacy scholars and professionals from a wide variety of geographical locations and 
backgrounds (n = 10). Because some of the information disclosed in the interviews could be sensitive (such as 
weaknesses in their assessment practices), pseudonyms were used instead of the interviewees’ names. The 
interviewees represented a wide range of geographical locations and professional backgrounds so a global 
understanding of assessment practices could be described. The sample consisted of participants from the 
United States, Australia, Canada, United Kingdom, Belgium, and Hong Kong. Seven interviewees were both 
teachers and researchers. One was the president of a large media literacy organization, another was primarily a 
field researcher, and yet another was a teacher. 

Instrument. A semi-structured protocol was developed to explore the views that the interviewees held 
regarding media literacy assessment. The survey instrument was developed after the analysis of the qualitative 
data. Based on the codes and categories identified during the qualitative phase, survey questions were 
developed. These survey questions were also guided by the literature on media literacy assessment. 
Interviewees mentioned numerous challenges they encountered in their work related to media literacy. 
Consequently, survey respondents were given a list of challenges that were identified by the interviewees. In 
order to validate and extend the qualitative findings, survey respondents were asked whether they encountered 
any of these challenges and to what extent they believe these challenges are important challenges to the field of 
MLE. The protocol was improved and validated via three expert reviewers. 

Data Collection and Analysis. The interviews were conducted online and recorded, using an online 
audio and video conferencing software application. Each interview lasted approximately 60 minutes. Interview 
transcripts were sent back to the interviewees for member checking. Each of the transcripts was coded twice. 
The first iteration identified more generalized codes and categories. The identification of themes was achieved 
during the second iteration. 

Phase 2: Quantitative Survey Research 

Participants. Survey respondents were also media literacy professionals and scholars from around the 
world. The specific strategy used to select participants was maximum variation sampling by contacting media 
literacy scholars and professionals from a wide variety of geographical locations (countries) and backgrounds 
(audiences the scholars and professionals work with, years of experience in the field, and their roles in MLE). 
The respondents were chosen based on these criteria as media literacy challenges may differ in countries 
depending on their educational systems and governmental and institutional regulations. In addition, 
professionals and scholars may have encountered different challenges depending on the audiences they teach 
or study, their role in the field, and years of experience. 

Snowball sampling was also used by asking respondents in the survey to provide contact information of 
others in the field and by dispersing the survey through media literacy associations. The survey was fully 
completed by 133 respondents and partially completed by another 38 respondents. Only data from the 
completed surveys were included in the analysis. The respondents represented 35 countries on five continents. 
The countries having the most representation were the United States (n = 30), Australia (n = 17), the 
Netherlands (n = 10), and Finland (n = 7). The respondents reported their roles in MLE, with some having 
multiple roles. Of the 133 respondents, 76 were teachers, 70 were researchers, 66 were developers of media 
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literacy instructional materials, and 40 reported having other roles in MLE. About 58% of respondents reported 
having multiple roles in MLE. On average, respondents who taught had 14.21 years of experience (SD = 

10.70). Respondents who conducted research had an average of 11.45 years of experience (SD = 8.80), and 
developers of media literacy instructional materials had an average of 11.14 years of experience (SD = 9.20). 
The respondents worked with a wide variety of audiences and integrated media literacy in a wide variety of 
subjects. 

Instrument. A web-based survey instrument was developed using the results from qualitative data 
analysis which provided guidance for areas of interest. After construction, the instrument was refined using 
input from three expert reviewers. 

Data Collection and Analysis. Survey data were collected in January and February of 2014. 
Respondents were kept anonymous, and it was not possible to track completion for the purposes of offering 
reminders to participants to complete the survey. The quantitative data was analyzed using SPSS software and 
some variables were recoded before they were analyzed. 

In the next section, we describe the main issues and challenges that emerged from the analysis of the 
qualitative interviews along with the presentation of evidence from survey data. This enables a demonstration 
of how these issues and challenges were also encountered by the survey respondents. Furthermore, we report 
the percentage of survey respondents who perceive these to be important challenges for the field of MLE, 
regardless of whether they personally encountered these issues in their educational practice. 

Results 

Eight main challenges were cited by the interviewees and validated by the survey respondents. Each of 
the challenges described below was mentioned at least ten different times in interviews by at least four 
different interviewees. 

Assessment Methods and Criteria are Context Dependent and Not Universal 

Because no one approach to measurement dominates the field, researchers and educators use various 
assessment methods and the creation of clear assessment criteria that are closely fitted to the particular learning 
context under investigation. Dr. Maas stated that media literacy assessment is generally not strongly 
developed: “It’s like it’s a hot topic but there isn’t that much empirical research. So it’s not that you can use 
some strong measurements which are accepted by a big community.” Dr. Maas explained further: 

So that’s definitely a challenge, that you don’t have any consensus or sort of shared views on good 
measurement tools and I think if you look at other fields within psychology you definitely have fields 
where there is some kind of agreement on how you should measure certain things. So that’s a big 
problem. 

Some MLE experts seek universal definitions and assessment instruments. Dr. Lin stated that to make media 
literacy a scientific academic discipline “we need to have a very concrete research method, research theory, 
and assessment standards. And we need a common view about what media literacy means, how it can be 
assessed.” She stated that there is currently not a common standard or systematic way to assess media literacy. 
This seems related to a more general question that Dr. Smith pointed out. She stated: 

So I think one of the challenges of media literacy scholarship is to ask ourselves: Is there a global 
media literacy? Does context specific knowledge help build toward a greater understanding of media 
that can occur regardless of the context in which we’re applying it? Or is it kind of bound a little bit by 
these major topics? And that I’m not sure, most of my own work has been bound by a particular topic. 
I’m not among the brave souls who’ve tried to say: These, whatever 20 items on a survey measure 
media literacy, regardless of what you’re talking about. 
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Similarly, when survey respondents were asked which media literacy challenges they encountered, 46% of the 
133 survey participants agreed that they personally found it a challenge that “media literacy outcomes are not 
explicitly defined, so it is not clear what should be assessed.” 

Although some scholars have attempted to develop a single media literacy assessment instrument, most 
interviewees mentioned that media literacy assessment is a context-dependent endeavor. Dr. Lin explained that 
media use is very culturally specific, and that media literacy in the United States therefore means something 
different than in Mainland China. Likewise, different age levels also require different assessments. Dr. 
Hammond stated that she could not commonly use the same coding scheme that she developed in a private 
school when working with a largely minority population. She explained: “We looked at them within their 
reference group, because you can’t really compare the children of Google executives to the children of factory 
workers on some kind of universal scale, that’s just not fair.” Dr. Smith also noted that her research team 
assesses MLE based on the educational context. For example, when they have a small number of students, they 
tend to assess MLE qualitatively, while if it is a large number, they tend to use mixed methods. 

A challenge related to this lack of research and agreement on assessment methods is the struggle to 
define what counts as valid practice and knowledge. Mr. Farrell explained: 

You can look at some of those videos that kids make and I can say: Wow that’s fantastic, that’s really 
sharp editing, but then somebody else might say: Yea but what do they actually leam from that? And 
argue, they may say what people have said to me: Aren’t you just valuing how far they can mimic an 
industry product? 

Dr. Davis similarly stated that notions such as creativity are very hard to pin down. He explained: 

High school kids love making horror films. So when they make that horror film, if they follow the 
codes and conventions perfectly, and make a really terrific horror film, you could ask the question, well 
what have they really demonstrated there? Have they demonstrated creativity because they simply 
followed someone else’s sort of template for a horror film? Or within following the template, have they 
shown some genuine creativity and ability and skill, and brought their own kind of sensibility and 
aesthetics to that. They’re not easy questions to answer really. 

Critical Thinking or Memorizing Content 

Though important to move beyond the assessment of lower order to higher order thinking skills, the 
interviewees did find this difficult. In addition to rules and regulations, there are pedagogical challenges as 
well. Dr. Lin described that students often mimic each other in her media monitoring exercise. After one group 
has presented their analysis, the second and third group will often follow the example of the first group. This 
keeps the students engaged only in lower level thinking. Similarly, Mr. Farrell explained that when he assesses 
an essay exam he may initially think that students can understand and analyze a text really well “but actually 
then you read 10 more essays which are pretty identical and you realized all they’ve actually done is remember 
or learn what the teacher has told them.” Overall, 25% of respondents stated that they found it difficult to move 
beyond assessing lower order thinking and content knowledge. 

Issues of Interpretation and Interrater Reliability 

Educators and researchers both struggle with evaluating students’ media literacy competencies. Even 
when trying to clearly define assessment criteria in rubrics, Mr. Farrell stated that it is still very relative and 
open to teachers’ interpretation how they then assess student work. He stated: “Even within that, it’s very very 
difficult, because there are criteria in there that say things like: excellent understanding or proficient or 
minimal and each of those things is only relative.” Because some theoretical and philosophical concepts such 
as creativity may be up for debate, different teachers often interpret these concepts differently, based on their 
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own background and beliefs. Dr. Davis stated that a teacher’s philosophy of media literacy relates to how they 
make judgments. A teacher with a more protectionist stance may evaluate students’ answers differently than 
someone with a more progressive stance. Mr. Farrell also explained that teachers and examiners often take 
their own perspective to MLE, and that their philosophical orientation may then sometimes determine the way 
they interpret student work: 

I think if you come to a sort of Len Masterman kind of way saying: I want them to unmask dominant 
ideology or something, then you end up kind of looking for that, and if you don’t see that in the work, 
then you think the work is worth less. Not worthless, but worth less than another piece of work. And I 
think that’s quite difficult, to try to kind of be reflexive about those things and to sort of shake that off 
and to say: Okay, I got to put aside my own personal preferences and say well: What are our criteria 
here and how are they meeting them? 

When teachers interpret students’ work differently and do not agree on what counts as quality, 
researchers would say that the interrater reliability is low. Both Mr. Farrell and Dr. Davis serve on examination 
boards in their respective countries (U.K. and Australia). They both stated that a lower interrater reliability 
seems to happen more in evaluating media production work rather than media analysis work. Mr. Farrell stated 
that his team changed grades in about half of the work, even though it was a relatively small change in many of 
the cases. However, he also stated that “you do get a certain proportion, probably about 10% of schools where 
the changes are massive, like 25% of the mark or more and that’s a real problem because you just see they 
don’t understand the standard at all.” Dr. Hammond whose team assessed media analysis work said that 
interrater reliability was sometimes also a problem in her research that required coding, but that it was not so in 
more than 10% of the cases. Several issues related to interrater reliability were also challenges for the survey 
respondents. Of 133 respondents, 36% found that “it is difficult to take your own philosophical views out of 
assessment and to think about the influence of your own background on the way you assess students.” In 
addition, 34% found it a challenge that “different teachers often score the same students’ work differently.” In 
addition, 23% of respondents found that “comparing the scores of one class or school to the scores of another 
class is difficult.” 

Ambivalence about the Quantification of Media Literacy Learning 

Participants struggled with the challenge of quantifying the variety of competencies involved in MLE. 
Some were frustrated and pessimistic about the inability to develop assessment measures. For example, Dr. 
Shennan expressed that the field needs to find a way to show that MLE works: 

If we can’t figure out how to assess this and really start racking up a body of research that shows that it 
works, then again, I think we’re lost. I think that in today’s world in particular, assessment is the name 
of the game. 

Ms. Brown stated that politically, one has to produce evidence that students have learned something. However, 
she explained that “producing evidence that makes sense to other people who probably don’t necessarily know 
anything about what you’re trying to do, and may be very skeptical about it is really really hard.” Dr. Lin also 
agreed that media literacy learning is very hard to quantify. Using more quantitative and standardized 
assessments to achieve this has not worked well for some interviewees. Mr. Farrell stated that only certain 
types of skills get assessed in standardized tests. He stated: 

And actually my experience of many years of actually assessing of both coursework and exams, I feel 
that exams end up just benefiting a very small proportion of kids who can kind of jump through the 
right hoops in a way. . . . They’ll memorize what they need to memorize and they’ll then knock it out. 
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One particular challenge with formal exams, according to the interviewees, is that they often allow for 
only one correct answer, or an answer that can be computer scored. Integral to media literacy, however, is the 
notion that a variety of responses may be correct. Dr. Smith stated that “sometimes the answer is ‘it depends.’ 
And the most media literate answer in my opinion would be: It’s not entirely right or entirely wrong but 
somewhere in the middle. And that’s why sometimes the quantitative items can be tricky.” Dr. Shennan also 
explained: 

And the main challenge I see is that since it really is a pedagogy and not a content area, that we’re not 
assessing what vocabulary do you know particularly. And therefore it doesn’t lend itself to convergent 
questions like multiple choice questions or, in fact, fundamental to media literacy is that different 
people may interpret the same message differently. So you could all have right answers that are all 
different. So it doesn’t map onto our standard way of assessing things, where everybody comes to the 
same answer, and so therefore I think that’s a bit of a challenge. 

She stated that lower order thinking skills can be assessed quantitatively, but that she has not found a 
quantitative instrument that effectively measures higher order skills. Dr. Hammond agreed, stating her lack of 
interest in multiple choice assessments as they measure lower order skills, such as recognition, rather than 
higher levels of thinking. Similarly, 45% of the survey respondents believed that “formal assessments (such as 
multiple choice questions or written exams) may not capture true learning.” 

Lack of Teacher Preparedness and Teacher Training 

A lack of training also seemed to contribute to an inability to move beyond the assessment of lower 
order thinking skills. Ms. Brown explained that when observing classrooms, she would sometimes notice that 
when children went off on a tangent, asking interesting and difficult questions, teachers were often not 
prepared to answer these questions and therefore would not know how to respond. In addition, Dr. Davis 
mentioned that the Australian primary school curriculum will soon include Media Arts as part of the arts 
curriculum. Even though primary school teachers will then be expected to teach and assess media literacy key 
concepts in their classes, these teachers are generally not trained media teachers which may make it 
challenging for them to actually teach and assess media literacy. MLE is also often expected to be taught and 
assessed using an integrated cross-curricular approach. For teachers who lack any media literacy training, Dr. 
Davis stated that it is often difficult to understand how to assess media literacy learning: 

We had a year four class where they made a film about a scientific process and it was a big task to get 
the teachers to move beyond just assessing the students’ understanding of the scientific process and to 
get them to understand how they could assess the students’ ability to shoot the shots well, to edit it 
together well, to record a voice over effectively and so on. 

Ms. Brown experienced challenges with English teachers assessing children’s media learning. Rather than 
assessing media learning they would assess content knowledge. She explained: “They were always saying: 
Well, media education is really really useful for improving kids’ attainment in English. I said, but we don’t 
want to know that. We know that happens, but what does their media learning look like?” Without proper 
training and assessment models, media teachers will only focus on very basic use of media and basic 
assessment techniques in their classes. A lack of preparedness and training was the most commonly 
encountered challenge by the survey respondents with 53% stating that they personally encountered this 
problem. 

Lack of Control over Assessment 

In addition to professionals avoiding certain styles of assessments, another challenge serves to limit 
their options as well. In some countries, there are typically standards and regulations imposed by institutions 
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such as the Ministry of (Public) Education or Department of Education. Mr. Farrell stated that when writing a 
specification for media studies in the U.K., he always has to work within those types of rules. For example, he 
is required to include timed written exams in the assessment specification of a particular secondary media 
studies course. He expressed frustration with a new education minister as he explained: 

He’s changing a lot of things and so much of it goes completely against the way media educators think. 
And one of the things he’s trying to cut out altogether is coursework. So everything will be based on 
tenninal exams, which I think, if that happens with media, it would completely ruin the experience for 
students, for so many students get so much out of all of the elements of the production work. 

He stated that the system has become too oriented towards results, exams, and measuring particular skills. Ms. 
Brown also expressed her frustration with the government in the U.K. She stated that: 

They have their own agenda about what should be learned. They’re interested in how kids get to learn 
that stuff and how can they get in the sort of the PISA [Programme for International Student 
Assessment] studies, how can we make our country come top of the table, international competitors and 
all this kind of rubbish, and really, you say, the kids are doing really well in media education and they 
say: Yes, so what? You know, who cares? I don’t. 

Dr. Hammond also expressed her frustrations with the U.S. educational system. She stated: “One of the 
reasons we’re afraid of assessment is that in the U.S., hyper-individualization of assessment has led to this 
really awful situation in our education system, where teachers, where everything depends on the score that the 
child gets.” Dr. Hammond stated that she is more interested in contextual and situational assessment of media 
literacy. Dr. Smith also expressed her concerns with high-stakes testing. She described a typical school day 
that is focused on teaching to a mandated, high-stakes test, which leaves little room for MLE and is currently 
not included in these assessments. However, not all participants perceived institutional control as a challenge 
as only 15% of respondents stated that they “feel very limited when it comes to assessing media literacy due to 
outside influences (such as governmental decisions and other decisions out of their control.)” 

Quality Assessments are Expensive and Time Consuming 

Some interviewees stated that assessments enabled to capture media literacy learning and higher order 
thinking skills is possible, but that they were time consuming, expensive, or too complex to use in classrooms. 
Dr. Sherman stated that media literacy learning by young children cannot easily be assessed, which makes it 
time consuming: “When we assess our stuff with 5-6-7 year olds, it means it has to be an individual interview 
and therefore it’s going to take more time to assess.” Dr. Lin, who uses multiple assessment methods stated: “I 
have 41 students in my class. So it takes me a lot of time to go through the examination papers and the 
assignments each time.” Ms. Brown mentioned that it is also a management challenge. There were typically 
too many children to assess, which therefore makes the assessment a blunt tool. She stated: “If you could just 
spend the whole time doing classroom observation all year long, let somebody else do the teaching, that would 
be fascinating and amazing, but who’s going to pay for that?” 

Perfonnance-based measures where students’ analysis or creative work is directly assessed can be time- 
consuming. Dr. Hammond described her performance-based measure as a powerful tool to assess media 
literacy learning. However, she also explained that it does have disadvantages: “What I don’t like about that 
measure is that it’s really a time consuming measure because it has to be hand-coded and you have to have 
interrater reliability, you have to train coders. It’s a very expensive measure.” She concluded that the real 
challenge is: “how to develop a cost effective and efficient measure of media literacy without oversimplifying 
or trivializing the robust competencies that we’re trying to measure.” Correspondingly, 44% of survey 
respondents believed that “assessments capturing higher order thinking skills related to media literacy are time 
consuming, expensive, or complex to develop.” 
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Paradoxes of Control, Assessment and Critical Autonomy 

Some members of the MLE community find the measurement of media literacy to raise an inherent 
paradox associated with power and agency. According to Dr. Maas, there are scholars who may be very 
skeptical towards the idea of assessment for theoretical and political reasons. He explained: “If you address 
questions such as assessment, I think it’s important to realize that by itself assessment is quite, sort of, it’s not a 
neutral thing to say, ‘we are going to assess this.’” 

Some of the interviewees noted that the control issues embedded in the assessment paradigm often 
contradict with the critical autonomy that teachers want to develop in learners. For example, Ms. Brown stated 
that she strongly believes in personalized learning and that people should be able to set their own learning 
goals and assess them with the help of others. However, pedagogically one often has to state general learning 
outcomes. She explained that written examinations or portfolios always create a situation in which people are 
teaching towards specific outcomes. She mentioned that “it cuts out the possibility that children’s learning 
might take a different route.” Similarly, Mr. Green stated that rubrics sometimes also control what students 
have to learn: “The rubric ends up dictating what they do, it becomes another way to steer them, and so, in fact, 
in some ways reinforces this idea that there’s this outside power over them which is exactly like what I want to 
get away from.” 

One of the goals of MLE is to give people a sense of power or agency in the world. However, when 
teaching towards specific outcomes, this moves power from the student to the teacher. The challenge is to find 
a balance between the control of the assessment and the sense of power and agency that teachers want for 
students. Dr. Smith discussed this contradiction between critical autonomy and control of assessment: “I think 
that the trickiness is then how to avoid being dogmatic and how to truly promote kind of active learning at the 
same time that you’re hoping that there would be sort of a deeper understanding.” Mr. Green also encountered 
this contradiction and explained: 

My issues with assessment are connected to a paradox in my teaching in general. I want students to 
take control of their learning, to actively create their own narratives. So, I set up situations in which I 
try to promote this free thinking, and I ascribe value to such thinking. In so doing, however, I am 
stealing power for myself and shaping what they do; that is, I am establishing that certain ways of 
thinking and being (open-minded, comfort with shades of grey) are being better than other ways of 
thinking and being (absolutist, black and white). It's the age old question: how do we oppose prejudice 
without exhibiting prejudice against people who are prejudiced? In teaching, how do we assess how 
open-minded someone is without ourselves being closed-minded about close-mindedness? 

Mr. Green values students’ narratives and conversations with them. Although he actively tries to shape 
and steer the story, he does not want to define the story, as assessment may do. In that sense, assessment can 
get in the way of storytelling by prescribing the story, and not naturally letting it evolve. This is similar to how 
Ms. Brown wants to leave the classroom open for other learning routes. In his words: “students’ narrative is a 
tender plant, and we need to be careful. Sometimes in an attempt to measure this plant, we break it. Or so it 
feels to me, clumsy as I am at measuring.” 33% of survey respondents also agreed that that “there is a 
contradiction between the control that is inherent to assessment and the development of autonomy in the 
learners.” 

Other Media Literacy Challenges 

Survey respondents identified challenges that were not identified by a preponderance of interview 
subjects. In the survey, respondents were asked to what extent they agreed with these less commonly 
mentioned challenges. One major challenge was the tension between student learning process and the media 
work products that students produce. Interestingly, 48% of survey respondents who assessed a media 
production piece could not fully observe media literacy learning by only looking at the product as they missed 
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the context in which it was produced. For example, instructors may not see the extent to which parents helped 
students develop their media products or used pre-existing templates. This was a challenge that only three of 
10 interviewees mentioned, but was clearly a challenge encountered on a broader scale. Another common 
challenge was identified by 38% of respondents who noted that self-assessment surveys may not measure true 
skills as people cannot judge their own skills with validity. Respondents also noted issues with expectations, 
with 35% of survey respondents concerned that students may what the teacher wants to hear rather than what 
they truly think or feel. A complete overview of the encountered media challenges by survey respondents is 
shown in Figure 1. 
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Table 1 

Perceived Challenges to the Practice of Media Literacy Assessment 


n 

Not a 
challenge 

A minor 
challenge 

Major 

challenge 

M 

SD 

There is a lack of teacher preparedness and teacher 
training to assess media literacy outcomes 

116 

15 

(12.9%) 

23 

(19.8%) 

78 (67.2%) 

2.54 

0.72 

When assessing a media production piece, you cannot 
fully observe media literacy learning by only looking at 
the product as you miss the context in which it was 
produced 

112 

20 

(17.9%) 

51 

(45.5%) 

41 (36.6%) 

2.19 

0.72 

Formal assessment (such as multiple choice questions 
or written exams) may not capture true media literacy 
learning 

110 

19 

(17.3%) 

41 

(37.3%) 

50 (45.5%) 

2.28 

0.74 

Media literacy outcomes are not explicitly defined, so 
it is not clear what should be assessed 

113 

30 

(26.5%) 

45 

(39.8%) 

38 (33.6%) 

2.07 

0.78 

Assessments capturing higher order thinking skills 
related to media literacy are time consuming, expensive, 
or complex to develop 

112 

19 

(17.0%) 

42 

(37.5%) 

51 (45.5%) 

2.29 

0.74 

Self-assessment surveys may not measure people's 
true skills as people cannot validly judge their own skills 

107 

14 

(13.1%) 

69 

(64.5%) 

29 (22.4%) 

2.09 

0.59 

It is difficult to take your own philosophical views out 
of assessment and to think about the influence of your 
own background on the way you assess students 

104 

30 

(28.8%) 

52 

(50.0%) 

22 (21.2%) 

1.92 

0.71 

Students often seem to say what 1 want to hear, rather 
than what they truly think or feel 

105 

37 

(35.2%) 

50 

(47.6%) 

18(17.1%) 

1.82 

0.71 

Different teachers often score the same students' work 
differently 

105 

23 

(21.9%) 

52 

(49.5%) 

30 (28.6%) 

2.07 

0.71 

I believe that there is a contradiction between the 
control that is inherent to assessment and the 
development of critical autonomy in the learners 

109 

33 

(30.3%) 

47 

(43.1%) 

29 (26.6%) 

1.96 

0.76 

It is difficult to go beyond the assessment of lower 
order thinking skills 

106 

40 

(37.7%) 

44 

(41.5%) 

22 (20.8%) 

1.83 

0.75 

It is very difficult to control learning environments in 
media literacy research and to therefore get valid 
quantitative results 

103 

33 

(32.0%) 

48 

(46.6%) 

22 (21.4%) 

1.89 

0.73 

Comparing the scores of one class or school to the 
scores of another class or school is difficult 

99 

31 

(31.3%) 

43 

(43.4%) 

25 (25.3%) 

1.94 

0.75 

1 feel very limited when it comes to assessing media 
literacy due to outside influences (such as governmental 
decisions and other decisions out of my control) 

103 

61 

(59.2%) 

30 

(29.1%) 

12(11.7%) 

1.52 

0.7 
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In addition to indicating assessment challenges encountered, respondents were also asked to gauge the 
extent to which they believe these are challenges for the field of MLE assessment. These findings are shown in 
Table 1. Not every respondent answered these questions, hence the varying sample sizes. The answers were 
congruent with the challenges the respondents personally encountered. For example, a lack of teacher 
preparedness and training to assess media literacy had the highest average rating (67.2% rated it a major 
challenge). This was also the challenge that most often encountered by respondents. Similarly, only 11.7% of 
the respondents felt very limited when assessing media literacy due to outside influences. This challenge also 
had the lowest average among respondents when asked if it was a challenge for the field of media literacy 
assessment. 

Survey respondents were invited to respond to an open-ended question to describe challenges that were 
not previously identified by the interviewees. There were only a few additional challenges mentioned here. For 
example, eleven respondents noted a lack of resources, such as equipment, technologies, examiners, and 
assessment materials. In addition, eight respondents mentioned a lack of interest in media literacy and its 
assessment by educational institutions, administrators, and colleagues, seven respondents mentioned the 
challenge of varying student skill levels and media experiences, and six respondents mentioned that media 
literacy is too broad and complex to measure. 

Discussion and Conclusions 

Many participants in this study expressed that the field of media literacy needs a way to show that MFE 
works. This would lend credibility to the field. This was also supported by Buckingham and Domaille (2009, 
26) who stated “There is no doubt that the absence of structured assessment procedures has contributed to the 
lack of status afforded to media education.” Although participants in this study recognized the need for valid 
and reliable research methods and instruments, many recognized that a single assessment or large scale 
assessment measures may not exist. This is attributed to media literacy outcomes, criteria, and assessment 
instruments being bound by the geographical, historical, and cultural contexts in which assessment takes place 
and in the context of the groups that are assessed (such as age and socioeconomic class). Similarly, many 
participants acknowledge that MFE does not lend itself well to quantitative and standardized tests, as they may 
not be able to capture the higher order skills that are integral to MFE. Interestingly, while many participants of 
the study have a desire to move beyond quantitative assessments to assess true media literacy learning (e.g. 
critical thinking, analysis, evaluation, and creative production of media messages), recent major international 
assessment efforts are mainly quantitative in nature (e.g. EAVI, 2010, 2011; UNESCO, 2013). 

Although the study’s participants have a strong desire to move beyond the assessment of lower order 
thinking skills and content knowledge, several challenges prevent them from doing so. Pedagogically it is 
difficult, and many participants mention that there is a lack of teacher preparedness and training in this area. In 
addition, some participants state that their choices in assessment methods are limited due to outside (typically 
regulatory) constraints which require them to use more standardized exams. Moreover, interrater reliability 
becomes a problem when assessing higher order thinking skills as an integral element of MLE is that there may 
not always be a single correct response or answer. Participants also agreed that it is not easy to move beyond 
assessing lower order skills and content knowledge because quality assessments that measure higher order 
thinking are expensive and time consuming to develop and use. One interviewee states eloquently that the real 
challenge is “how to develop a cost effective and efficient measure of media literacy without oversimplifying 
or trivializing the robust competencies that we’re trying to measure.” 

Recommendations for Future Research 

Based on the results of this study, several recommendations for research are offered. Several 
participants agreed that research on media literacy assessment is an underdeveloped area in the literature. 
Assessment instruments that measure higher order skills fundamental to MLE are needed. There is a need for 
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assessments to measure critical thinking and student ability to critically analyze, evaluate, and produce media 
messages. This may come in the fonn of multiple assessment methods aimed at examining various skill sets 
and different target audiences. It is also important that researchers and practitioners critically examine the 
existing assessments to determine whether they can measure the complexity and higher order nature of media 
literacy knowledge, skills, and attitudes that are integral to MLE. 

It would also be of benefit to study the extent to which countries differ in their views on media literacy 
assessment challenges. Though this study had participants from 35 countries, increased participation from 
within each country would be helpful. A study incorporating more voices from each country would offer a 
comparative look at differences across nations. 

Transfer of learning is another area that may benefit from more research. It is unknown whether media 
literacy knowledge, skills, and attitudes that are learned by students and assessed in specific educational 
contexts transfer to other external contexts. Although students may be able to critically analyze and evaluate 
media texts in the classroom, it is unclear whether they would also engage in this behavior when encountered 
with media messages outside of school. Furthermore, longitudinal research in this area would be of benefit. It 
is unknown if students retain their knowledge, skills, and attitudes after MLE ends. 

The literature provides anecdotal evidence on challenges that media literacy scholars and professionals 
may encounter. However, this is the first study to directly ask media literacy professionals about the challenges 
that they have personally encountered and to what extent they believe that these are challenges for the field of 
MLE. Given that this is the first study offering a comprehensive view of these media literacy assessment 
challenges from the viewpoint of professionals working in the field, it may serve as a vantage point for future 
research in this area and in working toward solutions in overcoming these challenges. 
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